DropDim: A Regularization Method for Transformer Networks

We introduceDropDim, a structured dropout method designed for regularizing the self-attention mechanism, which is key component of transformer. In contrast to general method, randomly drops neurons, DropDim part embedding dimensions. this way, semantic information can be completely discarded. Thus, excessive coadapting between different dimensions broken, and forced encode meaningful featureswith certain number erased. Experiments on wide range tasks executed MUST-C English-Germany dataset show that effectively improve model performance, reduce over-fitting, complementary effects with other regularization methods. When combined label smoothing, WER reduced from 19.1% 15.1% ASR task, BLEU value increased from26.90 28.38 MT task. On ST reach score 22.99, an increase by 1.86 points compared strong baseline.